Repository navigation
perf(arrow-select): zip a primitive array with a scalar without MutableArrayData - #11559
Conversation
|
run benchmark zip_kernels |
This comment was marked as duplicate.
This comment was marked as duplicate.
|
🤖 Benchmark completed (GKE) | trigger Target: zip_kernels Comparing zip-scalar-blend (deecbb7) to 4dc6bc2 (merge-base) diff Run configurationrun benchmark zip_kernels
env:
BENCH_FILTER: array_vs_non_null_scalar|non_null_scalar_vs_array
shards: 1
DetailsPer-runner informationzip_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-671s0p7k-bef850df-xlxs Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench zip_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
deecbb7 to
1137028
Compare
|
Rebased on #11625 |
|
run benchmark zip_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: zip_kernels Sharding factor: 1 (1 workers). Comparing zip-scalar-blend (03bef9f) to df4dc3e (merge-base) diff Run configurationrun benchmark zip_kernels
env:
BENCH_FILTER: array_vs_non_null_scalar|non_null_scalar_vs_array
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: zip_kernels Comparing zip-scalar-blend (03bef9f) to df4dc3e (merge-base) diff Run configurationrun benchmark zip_kernels
env:
BENCH_FILTER: array_vs_non_null_scalar|non_null_scalar_vs_array
shards: 1
DetailsPer-runner informationzip_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-144p8rlm-9d0d7b21-v79b Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench zip_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @mightsleep, the bot numbers for i32 look great, and the comparison test against zip_impl covers offsets, null masks and both sides well.
The description says this is part of #11437, but that issue is about merge, and its plan doesn't list zip. Should this PR get a checklist item on #11437, so the zip work is tracked alongside the merge PRs?
| 1 => blend::<i8, 1>(&from_array, &array, &scalar)?, | ||
| 2 => blend::<i16, 1>(&from_array, &array, &scalar)?, | ||
| 4 => blend::<i32, 1>(&from_array, &array, &scalar)?, | ||
| 8 => blend::<i64, 1>(&from_array, &array, &scalar)?, |
There was a problem hiding this comment.
What happens here for an Interval(DayTime) array whose values buffer is only 4-byte aligned? arrow-rs requires a fixed-width buffer to be aligned only to align_of of its native type (new_fixed_width), and IntervalDayTime is two i32s, so its alignment is 4. The Arrow format only recommends 8- or 64-byte alignment in memory.
For such a buffer (I built one by slicing a Buffer 4 bytes in), blend::<i64, 1> returns None, and the fallback panics in Buffer::typed_data, because MutableArrayData extends DayTime as i64. That panic also happens on main without this PR, so it needs its own fix.
For this PR, what do you think about blending DayTime as two i32 lanes? The aligned case would then never fall back, and as I read it, the alignment check in rows would no longer fail for any type that reaches blend. Could the test also add Interval(DayTime) to data_types, plus one case with a 4-byte-aligned buffer? random_primitive always produces aligned buffers. This needs IntervalUnit in the top-level arrow_schema import.
| 8 => blend::<i64, 1>(&from_array, &array, &scalar)?, | |
| 8 if *array.data_type() == DataType::Interval(IntervalUnit::DayTime) => { | |
| // arrow-rs aligns a `DayTime` buffer only to `align_of::<IntervalDayTime>()`, 4 | |
| blend::<i32, 2>(&from_array, &array, &scalar)? | |
| } | |
| 8 => blend::<i64, 1>(&from_array, &array, &scalar)?, |
There was a problem hiding this comment.
Thanks for digging into this and for filing #11862. I'd like to keep this PR about speed. The fast path dispatches on width only, and an unaligned buffer falls back to zip_impl through the generic alignment check in rows, so the behavior for it doesn't change here. Handling DayTime alignment in each kernel with a fast path would spread the same workaround around, while the fix in MutableArrayData, as #10171 did for MonthDayNano, covers zip, merge and every other kernel that copies through it at once. I can pick up #11862 once one of my open PRs lands, unless someone gets to it first.
| Some(make_array( | ||
| data.build().expect("the layout of the input array"), | ||
| )) |
There was a problem hiding this comment.
Could this return the error instead of panicking? zip already returns a Result, and PrimitiveScalarImpl in this file propagates its build error (line 633). If the function returns Option<Result<ArrayRef, ArrowError>> (line 171), the ? on blend keeps working, and the caller at line 149 becomes return output;.
| Some(make_array( | |
| data.build().expect("the layout of the input array"), | |
| )) | |
| Some(data.build().map(make_array)) |
There was a problem hiding this comment.
Makes sense, applied. It can't fail for a layout taken from a valid input, but propagating the error matches PrimitiveScalarImpl in the same file.
| // Every lane of a byte tests a constant bit of the same value: an AND and a compare of | ||
| // 32-bit lanes, which SSE2 and NEON have. Shifting the word by a different count per lane | ||
| // has no SIMD form before AVX2, and SSE2 has no 64-bit compare. |
There was a problem hiding this comment.
Did you check which instructions this compiles to? On rustc 1.99.1 release, I don't see an AND and a compare on either target. On aarch64, lane_masks uses the per-lane variable shift that the comment says has no SIMD form, plus scalar bit extracts:
ushl.4s v2, v1, v0 ; shift each lane by its own count
cmlt.4s v2, v2, #0 ; mask = lane < 0
sbfx w11, w8, #5, #1
On x86_64 (SSE2), it emulates the per-lane shift with multiplies, plus scalar shl/sar:
pmuludq xmm4, xmm3 # multiply by per-lane powers of two
psrad xmm8, 31 # mask = sign bit
Neither listing has a pcmpeqd, cmeq or cmtst. Both are branch-free, which is the property that matters here. Should the comment state that property instead, so that nobody later keeps the "AND and compare" form for a reason that doesn't hold?
| // Every lane of a byte tests a constant bit of the same value: an AND and a compare of | |
| // 32-bit lanes, which SSE2 and NEON have. Shifting the word by a different count per lane | |
| // has no SIMD form before AVX2, and SSE2 has no 64-bit compare. | |
| // Each mask tests a constant bit of one byte of the word. This compiles without a branch per | |
| // bit on x86_64 (SSE2) and aarch64. |
There was a problem hiding this comment.
my bad, thanks for checking. I had verified this loop for branches, not for how the masks are computed, and LLVM turns the compare into a shift into the sign bit, which SSE2 emulates with pmuludq. Took your wording.
03bef9f to
a477973
Compare
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @mightsleep for applying the earlier suggestions so quickly.
The description reports numbers for Decimal128 (the 2-lane blend::<i64, 2> path) and for a null scalar (the shared-buffer path), but add_benchmark in zip_kernels only has an i32 primitive generator, and bench_zip_on_input_generator has no array-vs-null-scalar case. Should those two cases be added to zip_kernels, in a commit before the change, so the benchmark bot can compare them against the merge-base and the numbers in the description can be reproduced from the repo?
…leArrayData Signed-off-by: mightsleep <acannix@proton.me>
a477973 to
2ac0674
Compare
|
Yes hello, added them in a commit before the change: decimal128 (an array against a scalar on either side, and against a null scalar) and array_vs_null_scalar for every type. Could you run the bot on zip_kernels @mbutrovich? |
|
run benchmark zip_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: zip_kernels Sharding factor: 1 (1 workers). Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
run benchmark zip_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: zip_kernels Sharding factor: 1 (1 workers). Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
run benchmark zip_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: zip_kernels Sharding factor: 1 (1 workers). Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
run benchmark zip_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: zip_kernels Sharding factor: 1 (1 workers). Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: zip_kernels Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
shards: 1
DetailsPer-runner informationzip_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-hr1hsdiv-b6316889-j9b7 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench zip_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: zip_kernels Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
resources:
arch: amd64
shards: 1
DetailsPer-runner informationzip_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-10zn6ylp-79de5834-b7fn Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench zip_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: zip_kernels Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
DetailsPer-runner informationzip_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-10zn6ylp-8d12e677-rxgh Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench zip_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: zip_kernels Comparing zip-scalar-blend (2ac0674) to 0debf10 diff Run configurationrun benchmark zip_kernels
baseline:
ref: 0debf10fc85c36c5fb01cbbb4cad3863d4497748
env:
BENCH_FILTER: from_(i32|decimal128)/(array_vs_non_null_scalar|non_null_scalar_vs_array|array_vs_null_scalar)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1
DetailsPer-runner informationzip_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-2afsrt1r-6e04f20f-xqf5 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench zip_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
oh wow 🤯 |
Jefffrey
left a comment
There was a problem hiding this comment.
should be good to go once merged up
| _ => return None, | ||
| } | ||
| } else { | ||
| array.buffers()[0].slice_with_length(array.offset() * width, len * width) |
There was a problem hiding this comment.
for the null scalar case, we could actually extend this to work for all datatypes (except runendencoded and unions) since we're just replacing the null buffer of the array; though we can do this in a followup PR
There was a problem hiding this comment.
yep, will do in a follow-up
# Conflicts: # arrow-select/src/zip.rs
|
Yes hello, merged up. |
|
thanks, some great numbers 🚀 |
Rich-T-kid
left a comment
There was a problem hiding this comment.
approving to remove from my queue (ignore)
Which issue does this PR close?
Rationale for this change
zipwith a scalar on one side extendsMutableArrayDataonce per scalar row and once per run of array rows, 6 to 7 ns a row on random masks. Callers include DataFusion's Spark%andpmod,unnestandunion_extract.What changes are included in this PR?
For a primitive array and a scalar of the same type:
i64lanes.Other types keep the
MutableArrayDatapath.Are these changes tested?
test_zip_primitive_with_scalar_matches_general_pathcompares againstzip_implfor 8 types (includingDecimal256andInterval(MonthDayNano)), with offsets, nulls in the array and the mask, a null and a valid scalar, on either side.ns per row, 8192 rows, a pool of 512 random masks so the branch predictor cannot learn them:
zip_kernels(i32) on GitHub runners (Neoverse-N2, Apple M1, EPYC 9V74 default and x86-64-v3), main / this PR: 22 to 60x on random masks, 1.07 to 1.35x when every row comes from the array.Are there any significant user-facing changes?
No.